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Outline  ^SFIT 

Air  Force  institute  of  Technology 

1  The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow  * 

•  DOD’s  Ongoing  Acquisition  Challenges 

•  Uncertainty  and  the  Need  for  Flexibility 

•  The  Need  to  Justify  Flexibility 

•  Challenges  with  Valuing  Flexibility/Capabilities 

•  Proposed  Valuation  Methodology 

•  Current  Expected  Value  Life  Cycle  Cost  (CEVLCC)  Model 
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W  The  Problem  4:SFIT 
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^ ■  The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow  * 

•  GAO  analysis  of  DOD’s  major  acquisition  programs* — 

•  69%  reported  an  increase  in  total  acquisition  costs 

•  Over  40%  of  these  programs  had  unit  cost  increase  of  at  least  25% 

•  On  average — 

•  R&D  costs  42%  higher  than  originally  estimated 

•  22  months  behind  planned  schedule 

•  The  older  the  program,  the  worse  the  trend 

•  Programs  in  development  >15  yrs  have  seen  an 
average  138%  increase  in  acq  costs,  and  over  36 
mos  of  schedule  delays 


*Source:  GAO.  2009.  DEFENSE  ACQUISITIONS:  Assessments  of  Selected  Weapons  Programs.  Washington,  D.C. 
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W  The  Historical  Solution  4:SFIT 

Air  Force  institute  of  Technology 

^ ■  The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow  * 


•  Forty  Years  of  Acq  Reform — 

•  Fitzhugh  Commission  (1971) 

•  DODD  5000.01  (1972) 

•  DODI  5000.02  (1975) 

•  OMB  Circular  A-1 09  (1976) 

•  DSB  Acquisition  Cycle  Task  Force  (1978) 

•  Defense  Resource  Management  Study  (1979) 

•  Carlucci  Initiatives  (1981) 

•  Nunn-McCurdy  Thresholds  (1982) 

•  Grace  Commission  (1983) 

•  Packard  Commission/Goldwater-Nichols  Act  (1986) 

•  Defense  Management  Review  (1989) 

•  DODI  5000.02  Revision  (1991) 

•  Federal  Acquisition  Streamlining  Act  (1994) 

•  Clinger-Cohen  Act  (1996) 

•  Intelligence  Reform  and  Terrorism  Prevention  Act  (2004) 

•  Weapon  Systems  Acquisition  Reform  Act  (2009) 
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The  Result 

The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow 

•  Little  to  no  change  in  acq  cost  growth  in  last  3  decades 

•  Desired  improvements  are  seldom,  if  ever,  realized 

•  Why? 

•  Reform  efforts  largely  aimed  at  cause  rather  than  symptoms 

•  Exactly  the  right  approach,  unless  root  cause  is  inevitable 

•  Resources  may  actually  be  squandered — 
better  off  mitigating  the  impacts 

•  Can’t  stop  earthquakes,  so  we  design 
earthquake-resistant  structures 

•  For  DOD  acquisition  programs, 
earthquake  =  uncertainty 

•  Increasing  rate  of  change,  and  increasing  system  complexity 


Air  Force  institute  of  Technology 
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Mitigating  the  Impacts 


4AFIT 
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The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow 


Every  major  program  must  contend  with  myriad  sources  of 
uncertainty 

Uncertainty  cannot  be  overcome 

Instead  of  tilting  at  the  windmill  of  uncertainty,  perhaps 
accept  uncertainty  as  a  fact  of  life,  and  explore  how  we 
can  design  systems  to  better  respond  to  it 
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Flexibility 

The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow 
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•  General  term  most  often  associated  with  ability  to 
effectively  respond  to  uncertainty  is  flexibility 

•  If  systems  can  be  designed  to  more  readily  respond  to 
sources  of  uncertainty/change,  impact  to  program  is  lessened 

•  Designing  flexibility  into  a  system  may  be  vital  to  achieving 
elusive  goal  of  improved  cost  &  schedule  performance 

•  Frequent  strategy  in  private  industry,  esp.  in  sectors 
characterized  by  high  rates  of  uncertainty/change 
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W  What  is  Flexibility?  4SFIT 
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The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow 

•  Explored  in  CSER  2011  paper 

•  “Defining  Flexibility  and  Flexibility-Related  Terminology” 

•  May  involve 

•  Design  flexibility 

•  Process  flexibility 

•  Adaptability 

•  Robustness 

•  Versatility 

•  Etc 

•  Specific  definitions  not  important  for  this  discussion 

•  “respond  effectively  to  uncertainty” 
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Investing  in  Flexibility  ^SFIT 

Air  Force  Institute  of  Technology 

The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow  * 

•  Current  DOD  acq  structure  would  makes  it  very  difficult 
to  invest  system  flexibility 

•  DOD  design  trend  is  toward  optimal  performance 

•  More  integrated,  point-solution  outcomes  less  capable  of 
responding  effectively  to  changing  requirements 

•  Can’t  justify  spending  money  without  a  verified  reqmnt 

•  Need  to  quantify  its  value  in  order  to  ascertain  when  and 
to  what  degree  the  investment  in  flexibility  is  worthwhile 

•  Need  rational  decision  making  methodology 
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\^Decision  Making  Under  Uncertainty-^^SPJT 
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The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow 


•  NPV 

•  Common  method  for  determining  time  value  of  money 

•  Used  in  some  studies  (&  some  companies)  to  value  flex 

•  Not  favored  by  researchers  for  decisions  involving  flex 

•  Not  effective  in  conditions  of  great  uncertainty  as  it  assumes  a 
predetermined  path  thru  an  established  set  of  alternatives 
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Decision  Making  Under  Uncertainty-^Sf  IT 

^ Air  Force  Institute  of  Technology 

The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow 

•  Real  Options 

•  Defined  as  the  right,  but  not  the  obligation  to  take  an  action 
at  a  pre-determined  cost  and  at  a  predetermined  time 

•  Preferred  approach  by  many  scholars 

•  May  not  be  suitable  for  DOD 

•  Black-Scholes  (underlying  financial  model)  requires  valued  asset  be 
traded  on  “efficient”  market,  w/  no  possibility  of  arbitrage 

•  Arguably  true  in  broader  capital  market;  not  likely  within  the 
DOD,  where  markets  are  often  artificial,  and  far  from  efficient 

•  B-S  model  assumes  random  fluctuation  of  price 

•  Dubious  premise  in  standard  market,  let  alone  DOD 
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V  Valuing  Flexibility  ^SFIT 

Air  Force  Institute  of  Technology 

1  The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow  * 

•  Fundamentally,  profit-centric  valuation  approaches  not 
suitable  for  DOD  systems 

•  The  ROI  is  a  political  or  military  outcome 

•  Incommensurable  units 

•  One  side  of  equation  is  cost  in  dollars 

•  Other  side  of  the  equation  is  military  outcome/capability 

•  A  flexible  system  does  not  have  intrinsic  value — 

•  The  capability  associated  with  that  flexibility  that  has  value 

•  To  assign  value  to  flex,  must  assign  value  to  military 
capability 
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Valuing  Capabilities  ^SFIT 

®  Air  Force  institute  of  Technology 

The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow  * 


•  But  how  do  we  ascertain  the  monetized  value  of  a 
military  capability? 

•  Fly  a  little  faster,  fire  round  a  bit  farther,  be  a  bit  more 
stealthy,  have  slightly  improved  reliability... 
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Valuing  Capabilities  ^SFIT 

®  Air  Force  Institute  of  Technology 

^ ■  The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow  * 

•  Willingness  to  Pay? 

•  Under  neoclassic  economic  definition  of  value,  item’s  value 
can  be  established  from  customer’s  willingness  to  pay 

•  In  theory,  value  of  a  particular  military  capability  could  be 
determined  from  the  maximum  amount  govt  is  willing  to  pay 
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V  Valuing  Capabilities  ^SFIT 

®  Air  Force  Institute  of  Technology 

1  The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow  * 

•  Use  budgeted  amount? 

•  Actual  system  cost  may  include  other  scarce  resources  not 
captured  in  govt  budget  (time,  critical  skills,  facilities,  etc) 

•  Need  to  account  for  opportunity  cost 

•  E.g.,  losing/vitiating  other  capabilities  by  virtue  of  this  investment 

•  Problem  becomes  recursive! 

•  Budgeted  amount  not  necessarily  max  govt  willing  to  pay 

•  Program  budgets  based  on  expected  actual  costs 

•  Budget  allocation  processes  notoriously  volatile,  unrelated  to  the 
merits  of  particular  program 

•  Defense  budgets  don’t  cleanly  map  to  capabilities 

•  No  budgeted  amount  for  non-baselined  reqmnts/capabilities 

•  Perceived  value  of  a  capability  may  vary  drastically! 
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Valuing  Capabilities  ^SFIT 

®  Air  Force  Institute  of  Technology 

The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow 

•  User  Query? 

•  Inherent  subjectivity 

•  Different  users  will  perceive  the  value  of  a  given  capability  differently 

•  Who  to  ask?  How  to  weight  responses?  How  to  reconcile  conflicts? 

•  End-users  often  not  conversant  in  the  language  of  budgets 
or  possess  meaningful  insight  into  costs 

•  Flexible  design  options  may  not  resonate  with  user 

•  Value  of  potential  capabilities,  vice  validated 

•  Many  ostensible  benefits  of  design  flexibility  may  be  of  great  value  to 
the  acquirer,  but  of  no  consequence  to  the  user 

•  Perceived  value  of  a  capability  may  vary  drastically! 
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An  Alternative  Approach  ^SFIT 

®  ®  Air  Force  institute  of  Technology 

^ ■  The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow  * 

•  LCC  as  a  proxy  for  value 

•  Refine  current  LCC  calculations  to  better  account  for  value 
of  capability  opportunities  likely  to  arise  in  life  of  a  program 

•  Though  uncertainty  not  deterministic — 

•  May  be  possible  to  employ  stochastic  probability  methods  that  can 
yield  more  accurate  cost  estimates 

•  More  accurate  LCC  estimates  (&  accompanying 
improvement  in  decision-making)  promises  enormous  ROI 
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LCC  Under  Uncertainty  ^SFIT 
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^ ■  The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow  * 

•  Key  Assertions* 

•  The  cost  to  develop,  procure,  &  operate  a  system  with 
some  assured  minimum  capability  over  its  lifecycle  is  not  a 
deterministic  value 

•  Instead,  this  cost  can  be  modeled  as  a  random  variable 
with  a  probability  distribution  resulting  from  a  set  of 
uncertainties  introduced  throughout  the  system's  life 

•  This  random  variable  metric  is  a  relevant  basis  for 
comparison  between  alternative  system  . . .  design  choices 

•  DARPA  only  considered  launch  and  on-orbit  failures 

•  Broader  applicability  not  explored 


‘Sources;  Brown  O.,  A.  Long,  et  al.  2007.  System  Lifecycle  Cost  Under  Uncertainty  as  a  Design  Metric  Encompassing 
the  Value  of  Architectural  Flexibility.  In  AIAA  Space  2007  Conference.  216-229;  Brown  O.  and  P.  Eremenko.  2008. 
Application  of  Value-Centric  Design  to  Space  Architectures:  The  Case  of  Fractionated  Spacecraft.  Wash,  D.C.:  DARPA 
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V/  Stochastic,  Dynamic  LCC  ^SFIT 
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1  The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow  * 

•  Expand  LCC  under  uncertainty  idea  to  a  robust  and 
comprehensive  methodology  for  effectively  valuing 
various  system  design  alternatives 

•  Extend  to  other  sources  of  programmatic  uncertainty 

•  New  threats,  technological  setbacks/breakthroughs,  reqmnt  creep, 
test  failures,  budget  fluctuations,  market  volatility,  etc. 

•  Apply  to  lower-level  design  decisions 

•  Dynamic  vice  static 

•  Continually  updated  decision  analysis  tool 

•  Current  Expected  Value  of  Life  Cycle  Cost  (CEVLCC) 
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W  CEVLCC  Methodology  4:AFIT 
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^ ■  The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow  * 

1 .  Establish  system  design  options 

2.  Construct  time-phased  PDFs  associated  w/  all  existing  key 
cost,  schedule,  &  tech  performance  parameters  of  program 

3.  Estimate  costs  associated  with  mods  (consistent  w /  PDFs)  to 
baseline  cost,  schedule,  &  tech  performance  parameters 

4.  Assign  time-phased  probabilities  for  potential  new 
capabilities  of  the  system 

5.  Estimate  costs  associated  w /  the  addition  of  new  capabilities 

6.  Calculate  standard  (i.e.,  traditional)  LCC  estimate 

7.  Calculate  CEVLCC  for  each  system  design  option  and  select 
alternative  with  the  lowest  CEVLCC 
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Marginal  Probability  Costs 

The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow 


fMFTf 
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Threshold 

Parameter 
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W  CEVLCC  Equation  4:AFIT 

®  Air  Force  institute  of  Technology 

1  The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow  * 


Marginal  Probability 
Costs  for  Modifiable 
Capabilities 


m 


Marginal  Probability 
Costs  for  New 
Capabilities 


n 


CEVLCC =  LCC  +  bY 


i=l 


i=l 


i 


T 

Standard 

LCC 


J  l 


I 

Probabilistic  value 


The  system  that  is  the  best  value  is 
simply  the  one  with  the  lowest  CEVLCC 


b  =  weighting  foctor 
m  =  no.  of  modifiable  capabilities 
n  =  no.  of  new  capabilities 


Air  University:  The  Intellectual  and  Leadership  Center  of  the  Air  Force 

Aim  High. ..Fly  -  Fight  -  Win 


22 


Potential  Challenges  ^SFIT 
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1  The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow  * 

•  Nature  of  DOD  acq  may  be  more  chaotic  than  stochastic 

•  Would  prevent  accurate  predictive  modeling  over  a 
reasonable  time  horizon 

•  MPC  models  would  need  to  be  comprehensive/current 

•  May  be  overly  cumbersome,  and  investment  cost  likely  to 
outweigh  benefits  at  some  point  in  program  life 

•  Only  applies  to  foreseeable  sources  of  change 

•  Only  valid  to  compare  design  options  that  meet  threshold 
reqmnt  levels 

•  Does  not  entirely  sidestep  problem  of  valuing  capability 

•  Design  option  performance  greater  than  threshold  (but  less 
than  objective)  has  temporal  and  intrinsic  (to  user)  value 
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Summary 

The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow 
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•  Uncertainty  leads  to  cost/schedule  overruns 

•  Only  so  much  we  can  do  to  mitigate  uncertainty 

•  Need  to  be  able  to  respond  effectively  to  it,  i.e.,  have  flexibility 

•  Difficult  to  justify  the  required  investment  in  flexibility 

•  Need  to  quantify  value  of  flex  =>  quantify  value  of  capability 

•  Not  feasible  to  quantify  military  capabilities,  so  need  alternate 
approach  capable  of  evaluating  design  options  strategically 

•  Current  Expected  Value  Life  Cycle  Cost  (CEVLCC) 

•  Top-down,  intrinsic  value  model  based  on  familiar  notion  of  LCC 

•  The  need  for  capability  changes  in  a  program  arises  in  a 
stochastic  manner  that  can  be  modeled  &  incorporated  into 
continually  updated,  expected  value  model  of  total  program  cost 
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W  BACKUPS 

♦ 

The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow 
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CEVLCC  Assumptions  ^SFIT 

®  Air  Force  institute  of  Technology 

^ ■  The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow  * 

1 .  As  programs  mature,  there  will  be  unpredictable  deviations 
from  the  program  baseline  that  affect  the  system’s  LCC 

2.  An  improved  LCC  estimate  is  possible  thru  probabilistic 
modeling  of  the  stochastic  processes  that  cause  deviations 

3.  The  required  investment  cost  to  calculate  an  improved  LCC 
estimate  is  more  than  offset  by  the  value  obtained 

4.  Given  the  CEVLCC  cost  accounting  methodology,  as  long  as 
each  design  meets  all  of  its  threshold  requirements,  then  its 
relative  value  can  be  inferred  from  its  cost 
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Research  Steps  ^AFU 

®  Air  Force  institute  of  Technology 
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1.  Characterize  Accuracy  of  existing  LCCs 

•  Compare  predicted  costs  to  actuals 

2.  Build  &  Characterize  Default  CEVLCC  Model 

•  Determine  which  (and  to  what  extent)  MDAP  parameters 
have  statistically  significant  relationships  w/  actual  LCC 

3.  Build  &  Characterize  Program-Specific  CEVLCC 

Model 

•  Modify  Default  CEVLCC  Model  to  incorporate  MPCs  into  its 
prediction  algorithm(s) 

4.  Evaluate  Utility  of  CEVLCC  Models 

•  Assess  broad-based  utility  of  both  models  by  comparing  their 
cost  to  value  over  a  wide  range  of  usage  parameters 
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An  Alternative  Approach  ^SFIT 

®  ®  Air  Force  institute  of  Technology 

^ ■  The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow  * 

•  A  Better  Solution  Would  Be  Able  To — 

•  Establish  the  merits  of  a  capability  without  having  to 
explicitly  determine  its  value 

•  Inherently  assimilate  various  capability  concepts,  merging 
them  into  a  single  solution  space  effectively  responds  to 
uncertainty 

•  Being  comprised  of  concepts  already  familiar  to  the 
acquisition  community  (i.e.,  life  cycle  cost  and  risk 
analysis),  thereby  greatly  reducing  cultural  entry  barriers 

•  Having  a  simple  premise  and  an  intuitive  output  (i.e.,  cost), 
both  of  which  encourage  adoption  among  stakeholders 
across  the  acquisition  community 
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